Skip to content

Refactor: Migrate conformance tests to StorageTransport and enhance retry logic #2693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
thiyaguk09 wants to merge 15 commits into
base: node-18
Choose a base branch
from
Open

Refactor: Migrate conformance tests to StorageTransport and enhance retry logic #2693

thiyaguk09 wants to merge 15 commits into from
+1,488 −503

Conversation

@thiyaguk09
Copy link
Contributor

@thiyaguk09 thiyaguk09 commented Jan 12, 2026

Description

This PR refactors the GCS conformance test suite to use the low-level StorageTransport.makeRequest architecture instead of high-level SDK abstractions. This change is necessary to ensure that the x-retry-test-id headers are correctly propagated through the transport layer for fault-injection testing.

Key Changes:

  • Transport Refactor: Updated conformance-test/libraryMethods.ts (specifically saveResumable, lock, and createBucket) to use the low-level transport.
  • Robust Bucket Creation: Implemented "Create-or-Get" logic in createBucket to handle HTTP 409 (Conflict) errors, ensuring tests are resilient to non-empty bucket cleanup failures.
  • Dynamic Metageneration: Updated the lock function to dynamically fetch current bucket metadata before applying retention locks, resolving metageneration mismatch errors.
  • Header Normalization: Enhanced StorageTransport to consistently lowercase all response header keys, ensuring case-insensitive lookups regardless of the underlying header object type.
  • Retry Logic Improvements: Refined the shouldRetry logic to better handle malformed JSON responses and specific GCS error reasons like rateLimitExceeded.

Impact

  • Reliability: Fixes several "Scenario 1" and "Scenario 7" failures in the conformance test suite caused by high-level abstraction interference.
  • Consistency: Aligns the Node.js library with the expected GCS Resumable Upload protocol and error-handling standards.
  • Maintainability: Provides a more predictable path for injecting test-specific headers without modifying the core public API.

Testing

  • Unit Tests: Added comprehensive test cases in test/storage-transport.ts to cover shouldRetry branches, including network-level transients (ECONNRESET), malformed JSON strings, and idempotent method logic.
  • Coverage: Increased line coverage for storage-transport.ts from ~73% to over 90% by targeting previously uncovered error-handling blocks.
  • Integration Tests: Verified against the local conformance test-bench.
  • Breaking Changes: None. This refactor only affects internal test utility execution and private transport processing.

Additional Information

The refactor addresses specific issues encountered when running the suite in Node 18+ environments where gaxios and google-auth-library response structures (specifically Headers) behaved differently than in Node 14.

Checklist

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease
  • Appropriate docs were updated
  • Appropriate comments were added, particularly in complex areas or places that require background
  • No new warnings or issues will be generated from this change

Fixes #issue_number_goes_here 🦕

@thiyaguk09
conformance test fix
f80e973
@thiyaguk09
fix
506d2ec
@thiyaguk09
fix
d8e71c5
@thiyaguk09
phase 3 fixes
a172e3b
@thiyaguk09
fix
066b20c
@thiyaguk09
download fix
26d3124
@thiyaguk09
iamSetPolicy fix
dedbfa8
@thiyaguk09
fix 720 test cases
3142cb9
@thiyaguk09
fix: all 720 conformance-test cases
eb1fb6a 
All 720 test cases have been fixed, and the code has been refactored.
@thiyaguk09
revert storage changes
ec2bb4f
@thiyaguk09
Improve storage transport code coverage
27e0733
@thiyaguk09
refactor: transition to low-level transport for conformance tests
e40de7c 
- Replace high-level bucket and file calls with
storageTransport.makeRequest.
- Fix Scenario 1 failures by implementing "create-or-get" logic for
buckets.
- Resolve metageneration mismatch in lock() by dynamically fetching
metadata.
- Normalize header keys to lowercase in transport response processing.
- Increase unit test coverage for shouldRetry logic and error handling.
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. api: storage Issues related to the googleapis/nodejs-storage API. labels Jan 12, 2026
@thiyaguk09
fix: resolve auth issue and temp remove chunk-based resumable upload
9ef7c12 
- Fixed authentication headers/token exchange in the transport layer.
- Reverted to single-shot resumable upload to isolate Scenario 7
  failures while debugging mid-stream offset recovery.
@thiyaguk09
fix authentication issue and remove docs check
08872b5
@generated-files-bot
Copy link

generated-files-bot bot commented Jan 13, 2026

Warning: This pull request is touching the following templated files:

  • .github/workflows/ci.yaml - .github/workflows/ci.yaml (GitHub Actions) should be updated in synthtool

@thiyaguk09 thiyaguk09 marked this pull request as ready for review January 13, 2026 10:27
@thiyaguk09 thiyaguk09 requested review from a team as code owners January 13, 2026 10:27
@thiyaguk09 thiyaguk09 force-pushed the node18/conformance-test branch from 0d23d09 to ed49471 Compare January 14, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

api: storage Issues related to the googleapis/nodejs-storage API. size: xl Pull request size is extra large.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thiyaguk09